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I. INTRODUCTION 

Models of attractor neural networks for processing se- 
quences of patterns, as a realization of a temporal associ- 
ation, have been of great interest following Hopficld's pio- 
neering work [1-7] and renewed interest has come through 
both, the availability of new theoretical dynamic ap- 
proaches to study the evolution of disordered systems, 
in particular neural networks near saturation [8-12] and 
experimental findings. Among the latter are the results 
of Miyashita et al. who showed that serial positions of 
stimuli to which monkeys were exposed during training 
are converted to spatial correlations of neural activities 
00. 

A neural network model with a symmetric learning rule 
that consists of a Hebbian part and a pair of pattern se- 
quences has been proposed to interpret the experiments 
[lol Ha ] . The analysis of the equilibrium states for both 
finite and extensive loading of patterns, revealed the pres- 
ence of correlated attractors with a decreasing correla- 
tion as the separation of the patterns in the sequence 
increases, in apparent support of the experimental work. 
Phase diagrams were obtained where correlated attrac- 
tors compete with Hopfield-like attractors and with sym- 
metric states ^3 01- The dynamical evolution to the 
stationary states has been studied recently by means of 
dynamical replica theory |l9j . 

Neural network models for sequence processing with 
asymmetric interactions are more natural from a biologi- 
cal point of view. Due to the lack of microscopic detailed 
balance, however, the equilibrium states of the network 
cannot be obtained by means of the statistical mechan- 
ics approach [2jj, and one has to resort to a dynamical 
study. Analytic and numerical studies of the stationary 
states and some aspects of the dynamics for the com- 
petition between pattern reconstruction and asymmetric 
sequence processing in the case of finite loading of pat- 
terns, that is in the absence of stochastic noise, where 
the ratio a = p/N between the number of patterns p 
and units N is zero, appeared in works by Coolen et 



al. pH 0] ■ Phase diagrams for the stationary states of 
a parallel dynamics exhibit cither stable Hopfield-like or 
symmetric mixture states characterized by fixed-point so- 
lutions and stable limit cycles described by periodic fixed 
points. Non-stationary solutions of the dynamics also ap- 
pear for increasingly large number of patterns [2lj . More 
recent works deal with stationary limit cycles in asym- 
metric sequence processing without pattern reconstruc- 
tion, for extensive loading of patterns [12,23-26]. 

It is important to consider the effects of extensive 
loading of patterns and to study the stability of the 
states that appear in the competition between pattern 
reconstruction and asymmetric sequence processing to 
the presence of stochastic noise (finite a). This noise 
could be due to the presence of a previously learnt macro- 
scopic number of patterns following a specific learning 
rule. Since the problem is a dynamical one, it is also 
important to study the dependence of the network per- 
formance on initial conditions. 

Those are issues that, apparently, have not been con- 
sidered before and the purpose of the present paper is to 
study them on a tractable dynamical feed-forward lay- 
ered neural network model with no feed-back loops |27j . 
This is an extensively used model that consists of identi- 
cal layers of N non-interacting units on each layer, with 
synaptic interactions only between units on consecutive 
layers. The feed-forward nature of the model and the 
updating of the units on each layer endow the network of 
a dynamics in which the layer index becomes a discrete 
time. We consider an interaction matrix that yields ei- 
ther a static Hebbian noise or a dynamic Hebbian plus- 
sequential noise and restrict the work to binary units and 
patterns. In the finite loading case, the model becomes 
identical to that of Coolen and Sherrington for a parallel 
dynamics 0|. 

The outline of the paper is the following. In Sec. 2 
we present the model and the relevant order parame- 
ters. In Sec. 3 we derive the macroscopic dynamics of 
the network and in Sec. 4 we present the results in the 
form of phase diagrams of stationary states and regions 



2 



of non-stationary solutions. We end with a discussion 
and conclusions in Sec. 5. 



II. THE MODEL 

The network model consists of L layers with N binary 
Ising units (neurons) on each layer I in a microscopic 
state <r(l) = {<7i (I), . . . , ctat(Z)}, in which each ai(l) = 
±1. The state +1 represents a firing neuron and the 
state —la neuron at rest. The microscopic dynamics of 
the network is generated as follows: given a configuration 
on the first layer, <r(l), all units on layer ^ + 1 are updated 
simultaneously according to the alignment of each unit i 
to its local field 



N 



(i) 



due to the states of the units on the previous layer 
following a stochastic law with probability 



Prob(oi(l+l)|<r(l)) = 



exp[/M(l+l)hi(l + l)] 
2cosh[ ( 3h i (l+l))] 



(2) 



Thus, the network has a parallel dynamics with no feed- 
back loops in which the layer indices may be associated 
with discrete time steps. Here, Jij{l) is the synaptic con- 
nection defined below between unit j on layer / and unit 
i on layer I + 1. The parameter /3 = T^ 1 controls the 
synaptic noise such that the dynamics of the network be- 
comes a deterministic one when T — > and fully random 
when T — > oo. In the former case, the state of a unit is 
given by the deterministic form 



<Ti(l + 1) =sgn[fti (/ + !)] 



(3) 



A macroscopic set of p = aN independent 
and identically distributed random patterns ^{l) = 
{#(!), •■■.&(0}. H=l,...,p, and each = ±1 

with probability \ are stored in the learning stage on the 
units of each layer I, independently of other layers. Since 
we are interested in the retrieval of one or a small number 
c of patterns and in the recognition of a finite sequence, 
we assume a learning rule of the form 



J «w = jf E tfa+iH*#(o ( 4 ) 

H,p=l 

+ 7j E £^ + 1)^(0 , 



H,p=c+l 



where 



B u 



vSpp + (1 - i^Spp , \i mod c 
&<W + 0- _ b ) s w > M mod P 



(5) 



in which B^ p is non-zero only for fi > c+1. These are the 
(layer independent) elements of matrices A and B, with 



continuous independent v and b (0 < v, b < 1), in which 
<5 MP = 1 if /i = p, and zero otherwise, while S pp = S p p+i 
is a permutation matrix element. The signal-to-noise 
analysis carried out below depends on the assumption 
that the patterns can be separated into c "low" or con- 
densed ones, with finite overlaps with the states of the 
network in the N — > oo limit and (p — c) " high" or non- 
condensed patterns with overlaps of 0(l/y/~N) with the 
states. The assumption is justified in the case of a block- 
matrix interaction (here, a diagonal two-block matrix) 
and the specific form we choose is inspired in an earlier 
proposal [181 . In that spirit we assume here not necessar- 
ily the same weights, v and b. This is in order to contem- 
plate different simple possibilities and to avoid an even- 
tual closure problem in the noise term discussed below. 
The first part of Jij(l) represents a finite cycle which is a 
superposition of a Hebbian learning (the diagonal part of 
A) of single patterns and of patterns in a sequence (the 
non-diagonal part). Again, in the spirit of that reference, 
one may think of the condensed part in the presence of an 
infinite cycle (in the large p limit) of a previously learnt 
similar superposition of Hebbian and sequential patterns 
represented by the second part of Jij(l), which acts as a 
noise. Eventually, one could have < v < 1 and a Heb- 
bian noise with 6=1 which is a rather convenient 
choice. 

The learning stage of the network is a dynamical pro- 
cess that involves patterns on two consecutive layers. The 
Hebbian part of the rule may be thought as a static pro- 
cess that reinforces the learning of the same pattern on 
every pair of consecutive layers (times) whereas the se- 
quential part of the rule is a dynamic process in which 
the synaptic interaction is due to a pattern at a given 
time with the following pattern in the sequence at the 
next discrete time. It has been found before, in studying 
the finite loading case, that the competing static process 
has a stabilizing effect on the dynamic sequential pro- 
cess leading to a phase of symmetric states, which are 
the only ordered stable states within a wide range of rel- 
ative synaptic strengths under an appropriate amount 
of synaptic noise parameter T. Eventually, the static 
process may fail to lock the transitions in the dynamic 
process and non-stationary quasi-periodic solutions may 
appear which have already been found in the case of fi- 
nite loading (2l| . These are natural features of the model 
that are enhanced by stochastic noise (finite a) as will be 
seen and discussed in this work. 

The properties of A, in particular its eigenvalues and 
the complete set of orthogonal eigenvectors, lead to sym- 
metries of the solutions of the non-linear parallel dynam- 
ics in the form of a v/(l — v) duality, that has been dis- 
cussed for finite loading of patterns |2j . We find here an 
additional b/(l — b) duality. For the case of continuous 
bifurcations of solutions, the eigenvalues of the matrix 
yield the transition temperatures to the ordered states 
and the eigenvectors give the symmetry directions of the 
macroscopic overlaps between the states of the network 
and the patterns. 
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Wc define the macroscopic overlap between the config- 
uration tr(l) of the network on layer / and one or more 
condensed key patterns £^(1), fi = 1, . . . , c, on that layer 
as the large-N limit, m M (Z), of 

1 N 

<(o = ^E^mo) , (6) 

where the brackets denote a thermal average with Eq.(2). 
Since the number of condensed patterns is finite, one 
may use the self-averaging property to write m A '(/) = 
(£f (0( CT i(0))i wner e the outer bracket here and below 
denotes a configurational average over the patterns. Sim- 
ilarly, we define the overlap between the same config- 
uration and a given non-condensed pattern p. = 
c + 1, . . . ,p, on layer /, 

1 N 

M &(o = ^E^;(o ■ (7) 

Assuming that a given configuration of the first layer 
has a finite overlap m^(l) = 0(1) with one or more 
condensed patterns and overlaps MjJ-(l) = 0(1/ VN) 
with the non-condensed patterns, the dynamic evolution 
of the network will yield overlaps m M (Z) = 0(1) and 
Mx(l) = 0(1/ >/N) on the following layers. We consider 
next the evolution equations for the overlaps. 

III. DYNAMICS OF THE NETWORK 

Adapting the standard procedure for the layered net- 
work to our model, wc write the local field at a unit on 
layer I + 1 due to the overlaps with all patterns on layer 
I, in the large-N limit $2$, 

C 

h(l + 1) = E + l ) A w mP ( l ) + Z S) , (8) 

where the first term is the signal and Zi(l) is the large-N 
limit of the noise 

Ri{l)= E + 1 ) B »p M nV) . (9) 

fi.p=c+l 

due to the overlaps of the states with the non-condensed 
patterns. This is a random quantity in both the patterns 
on layer I + 1 and the implicit dependence on thermal 
and configurational randomness of the overlaps M^(l) 
on layer I due to the previous layers. 

The noise gives a finite contribution to the local field. 
Indeed, due to the fact that M&(1) = 0(1/ VN) and that 
Ri(l) is a sum of a large number of statistically indepen- 
dent random variables, one can first apply the central 
limit theorem and then the law of large numbers to con- 
clude that Zi(l) follows a Gaussian distribution with mean 



zero and a variance A 2 (I) given by the large-N limit of 

A^(0= E «[6M^(0 + (l-6)M^- 1 (0] a )> • (10) 

We used the fact that the patterns are unbiased and un- 
corrected random variables with (£f(Z + 1)) =0 and 

+ !)£?(* + !)> = */«'• 
We make use now of the local field to derive the recur- 
sion relations for the vector overlaps with the condensed 
patterns, m(l) = {m M (7)}, fj, = 1, . . . , c, and for the vari- 
ance of the noise. For the former we obtain 

m(l+l) = (i(l+l) J Dz tanh{/3[£(m).Am(Z)+A(Z)z]}) , 

where Dz = e~ z ' 2 dz/\/2ir and the brackets denote an 
average over the explicit patterns. The variance of the 
noise requires recursion relations not only for the av- 
erage squared non-condensed overlaps, ((M^(l) 2 )) and 
((M^ -1 (I) 2 )) , which can be derived in the usual way |27j . 
but also for the correlation of two consecutive overlaps 
((M^j(l)M^~ l (I))) . This generates, in turn, correlations 
between next-to-consecutive overlaps, ((M^(l)M^~ 2 (l)}} 
and so on, which requires to keep track of a general form 

Cl{l) = E (([bM» N (l) + (l-b)M» N -\l)] (12) 

p=c+l 

. [bM£- n (l) + (l-b)M£- n -\l)})) 
leading altogether to p — c + 1 recursion relations, 

A 2 (Z + 1) = b 2 (a + (3 2 I 2 A 2 (l)) (13) 

+ 2b(l-b)(3 2 I 2 Cl(l) , 

C?(i+1) = b 2 /3 2 I 2 C 2 (l) (14) 

+ b(l-b)[a + (i 2 I 2 (A 2 (l)+C 2 (l))] , 

C 2 n (l + 1) = b 2 [3 2 I 2 C 2 n (l) (15) 

+ b(l-b)p 2 I 2 (Cl_ 1 (l) + C 2 n+1 (l)) , 

for n = 2, . . . ,p — c — 1 and C p _ c = A, where 

b 2 = b 2 + (l-b) 2 (16) 
1(1) = l-q(l) . 

Here, q(l) = ((cr(l)) 2 ) is the spin-glass order parameter 
given by 

q(l) = (J Dz ttmh 2 {f3[£(l + l).Am(l) + A(l)z}}) . (17) 

Note that the Gaussian noise is symmetric under the 
change b — > (1 — 6) and that in the case where either b 
is one or zero, that is for purely Hcbbian or sequential 
noise, respectively, the recursion relation for the variance 
reduces to the simple form 

A 2 (l + l) = a + p 2 I 2 (l)A 2 (l) . (18) 
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Otherwise, one has to face the full set of recursion rela- 
tions which may become quite a task for an asymptot- 
ically large p. Indeed, the system of relations may not 
form a closed and finite set and there is no guaranty that 
this is not, in general, the case. Fortunately, by work- 
ing out numerically the equations for the dynamics in all 
the cases we were interested in this work we found that 
the set of recursion relations is practically finite and one 
proceeds as follows. 

Given an initial overlap m(l) = {m^l)}, /j, = 1, . . . , c, 
on the first layer, and taking A 2 (l) = C 2 (l) = a, for all 
n, Eqs. (11)-(17) describe the dynamics of the network 
and its stationary states. The latter are given by the 
fixed-point solutions m* = m(l+ 1) = m(l), A* = A(l + 
1) = A(l) and C* = C n {l + 1) = C n (l), for each n = 
1, .. . ,p— c— 1. Expressing Cj" in terms of all higher C*'s 
yields (A*) 2 oc a and in the case of b either one or zero 
this reduces to the simple form 



*\2 



(A*) 



l-/3 2 (l-# 



(19) 



where q is the fixed-point value of q(l). 

There may also exist other solutions, as non-stationary 
states, as will be seen in the next section. Solving for 
T = 0, searching for fixed-point solutions and for non- 
stationary states one finds that only a small number of 
the C*'s on the (p — c)-cycle of non-condensed overlaps 
are clearly non-zero and all others vanish. The ones that 
survive become smaller with increasing n indicating the 
vanishing of the averaged overlaps ((M^(l)M^~ n (I)}} . 

In the case of finite loading of patterns, where a = 0, 
the variance of the noise vanishes and the equations for 
the overlaps and the spin-glass order parameter become 
disconnected and we recover the equations for the over- 
laps found in previous work plj . 



IV. RESULTS 

We show and discuss next our results and use the so- 
lutions for finite loading as a guide. We are mainly in- 
terested in the network performance in the case of train- 
ing with a macroscopic number of patterns and on the 
impact of stochastic noise on the dynamics and the sta- 
tionary states. Among the important features are the 
critical loading level, the relative size of the regions with 
meaningful information processing, the presence of sta- 
ble symmetric mixture states and quasi-periodic non- 
stationary solutions. We also want to find out the specific 
dependence, if any, of these properties on the form of the 
stochastic noise and we restrict ourselves to Hebbian or 
Hebbian plus sequential noise. 

The phase diagrams discussed below are obtained from 
the dynamical equations for the macroscopic order pa- 
rameters and they depend naturally on initial conditions. 
Quite different sets of these conditions may reflect dis- 
tinct basins of attraction of the stationary states of the 
model (specified by a given pair of values of v and b) for 



given T and a. It will be seen that non-stationary states 
can be reached from certain initial overlaps. 

We concentrate on a small number of macroscopic con- 
densed overlaps, specifically c = 3 and c = 4, since the 
results depend on whether c is even or odd, with the 
remaining p — c patterns as noise in the case of exten- 
sive loading. This already illustrates the typical results. 
For specific results on higher values of c and the differ- 
ences between an even and odd number of patterns in the 
case of finite loading we refer the reader to earlier work 
[2ll | . Some of these results will be used below to infer the 
changes that one may expect for extensive loading. 



A. Finite loading 

First we reconsider the solutions for finite loading, 
where a — 0, in order to show the phases that appear, 
the periodic features of the cyclic phase and the quasi- 
periodic solutions in the region of unstable fixed points. 

The (T, v) phase diagram of stationary solutions for 
a = and c = 4 is shown in Fig.l for the best initial 
overlaps that favor single-pattern reconstruction (Hop- 
field or Hopfield-like solutions, see below). There is a 
paramagnetic phase (P) where m* = and q = above 
T = 1. This is a line of continuous bifurcations to 
a phase of stable symmetric fixed-point solutions (S), 
m\ = m*. = nig = m| ^ 0, that ends to the left at 
a boundary (dashed line) obtained analytically beyond 
which the symmetric states become unstable. This is a 
region of non-stationary quasi-periodic solutions marked 
q-p, as will be seen below, that is included with the warn- 
ing that it is not a true part of the phase diagram of sta- 
tionary states. The same applies to all other phase dia- 
grams in this paper where the q-p solutions are shown. 
It is worth noting that the symmetric states are the only 
stable states in the S region due to the stabilizing effect 
of the static condensed patterns in the learning rule and 
that this is a characteristic feature of the model, as dis- 
cussed in Sec. 2. Note that the symmetric states are 
enhanced by synaptic noise. 

Eventual alterations of the learning rule aimed at elim- 
inating what has been termed as " spurious" (here not at 
all) symmetric mixture states, say by means of the intro- 
duction of a bias in the distribution of the patterns [2^ , 
would leave nothing but the non-stationary q-p states 
over a large region of the phase diagram. 

The upper region of the phase diagram is a phase of 
stable Hopfield-like fixed-point solutions (H), with the 
four components to* , . . . , m% not all equal and different 
from zero, except at T = where one component may 
be one and the other ones zero. Using this zero T result 
as an initial solution we construct by numerical iteration 
of the overlaps the upper and lower phase boundaries 
indicated by solid lines. All other initial conditions yield 
phase boundaries above the upper or below the lower 
line and larger regions of q-p solutions. Those lines arc 
boundaries of first-order transitions for all values of 
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FIG. 1: Phase diagram for c = 4 condensed patterns for fi- 
nite loading (a = 0), with initial overlaps m\ = l,m2 = 
77i3 = 7714 = 0. Hopfield-like (H), symmetric (S), cyclic (C), 
paramagnetic (P) phases and region of quasi-periodic (q-p) 
solutions. Solid and dashed lines indicate first-order and con- 
tinuous transitions, respectively, here and below. 

except v = 1 or zero. The first-order transition is to 
the region of q-p solutions with remanent finite overlap 
components. The location of the transition for c = 4 is, 
as expected, close to that for higher c obtained in ref. 
|2lj . We also find, as expected, larger discontinuities at 
the first-order transition and shorter remanent overlaps 
in the q-p region, with our smaller number of condensed 
patterns than in that reference. 

The lower region of the phase diagram is the phase 
of period-4 stable stationary cyclic solutions (C). These 
are solutions in which m,i(l + 4) = rrii(l) for i = 
1, . . . , 4, with one overlap close to one and the other ones 
near zero in each time step after a transient. This fact 
characterizes sequence processing in which the network 
makes a transition from one pattern to the next at each 
layer. The upper and lower first-order phase boundaries 
appear symmetrically in the phase diagram due to the 
v j( \ — v) duality and both are equally sensitive to initial 
values for the overlaps. 

The nature of the cyclic and quasi-periodic solutions 
for a — and c = 4 is best illustrated by the power spec- 
tra shown in Figs. 2 (a) and (b), respectively. The first 
one is for a typical v = 0.1 and T = 0.15, within the phase 
of cyclic states, and the other one is for a typical v = 0.30 
and T = 0.35 in the region of non-stationary states. The 
power spectrum S((j), where u> is the frequency conju- 
gate to the layer index (a discrete time) may be defined 
in extension to the case of a continuous time p9l | , as 

1 L 

S(cj) = (const) limi^oo-l^e^'miCOl 3 - (20) 

1=0 

where mi(i) is any one of the components of the overlaps. 

The spectrum in Fig. 2a clearly exhibits a period- 
4 solution. In contrast, the spectrum of non-stationary 
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FIG. 2: Power spectrum S(lj), for c = 4, and a = 0. (a) 
Periodic 4-cycle for v = 0.1 and T = 0.15 in the C phase, with 
wo = 1.570796 and its three harmonics, (b) Non-stationary 
quasi-periodic (q-p) states for v = 0.30 and T = 0.35 in terms 
of four basic frequencies. 



solutions shown in Fig. 2b indicates quasi-periodic so- 
lutions. These are solutions characterized by main fre- 
quencies that are linear combinations of four recognizable 
basic frequencies. 

Consider next the (T, v) phase diagram of stable sta- 
tionary states for a = and c = 3, again with initial con- 
ditions mi = l,7?i2 = W3 = that favor single-pattern 
reconstruction, shown in Fig. 3. The symmetric states 
are now stable over the whole diagram, up to T = 1, 
except for an upper and a symmetrically placed lower re- 
gion of unstable states, where Hopfield-like (H) and cyclic 
(C) solutions, respectively as indicated, are the only sta- 
ble solutions. The stability of the symmetric solutions 
for all v at low T is a characteristic feature of the phase 
diagrams for an odd number of condensed patterns \2\\ . 
There is also a pair of tiny q-p solutions, that do not 
appear on the scale of the figure, between the exclusive 
H and C regions and the exclusive S phase for high T. 
The H or C solutions are stable everywhere above or be- 
low the upper or the lower solid lines, respectively The 
symmetric states compete for stability with Hopfield-like 
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FIG. 3: Phase diagram for c = 3 condensed patterns for a = FIG. 4: Phase diagram for c = 4, extensive loading 
with initial overlaps mi = l,m2 = = and phases (a — 0.008) and Hebbian noise, with initial overlaps mi = 
described in the text. 0.21, 7712 = 7713 = 7714 = 0.20. Spin-glass (SG) phase as indi- 

cated. 



or cyclic solutions in the regions H + S or C + S, de- 
pending on initial values of the overlaps. The symmetric 
states in the S phase are, again, the only stable states 
in the intermediate region between the two first-order 
transitions. Other initial more symmetric overlaps yield, 
again, smaller H and C phases and pairs of q-p solutions 
between the S and the H and C phases. 

B. Extensive loading 

For extensive loading, with a 7^ 0, one has to solve 
the full set of Eqs. (11)-(17). We still expect to have 
three ordered phases: H (Hopficld-like) , S and C, with 
specific m* ^ and q ^ 0. Also, a disordered spin- 
glass phase (SG) should appear with m* = and q ^ 
at the end of the ordered phases and this phase should 
survive for all higher T. The latter is a property of the 
layered network. All the transitions are now expected to 
be discontinuous and all the phase boundaries have to be 
obtained numerically. 

One may expect the v j(\ — v) duality also to hold for 
non-zero a. Indeed, the Gaussian noise due to the non- 
condensed patterns does not change neither under the 
set of linear transformations that keep the probability 
distribution of the condensed patterns invariant nor un- 
der the permutation index matrix S = {Sp P }. These are 
the two basic transformations which lead to a one-to-one 
correspondence between every state in the upper part 
(y > 1/2) in the (T, v) phase diagram and a state in the 
lower part [5l| . The duality serves to check the symmetry 
between the numerically constructed phase boundaries. 

The new feature of the (T, v) phase diagrams for both 
c = 3 and c = 4, not shown here, that are reached 
from Hopfield type initial overlaps that favor single- 
pattern reconstruction, 777.1 = 1,7772 = 7773 = and 
mi = l,m 2 = m 3 = m 4 = 0, respectively, is that the 
symmetric states become stable at low temperatures, in- 



cluding T = 0, within a finite range of intermediate val- 
ues of v as soon as a is non-zero. These states remain 
as the only stable states in that region, and there are H 
and C phases for large and small values of v, respectively, 
and a SG phase at high T. The phase boundaries to the 
H and C phases are nearly the same for both c = 3 and 
c = 4, and there appear no q-p solutions. The result- 
ing phase diagrams are very similar for either a Hebbian 
noise or full (Hebbian plus sequential) noise with b = v, 
the main difference being that the symmetric phase be- 
comes somewhat enlarged at high T in the latter case. 

Other more symmetric initial overlaps, which favor 
symmetric mixture states, lead to somewhat different re- 
sults for c = 3 and c = 4. For the latter, initial overlaps 
mi = 0.21,m2 = 777,3 = ™4 = 0.20 reduce the size of 
the H and C regions and lead now to intermediate re- 
gions of q-p solutions, as shown in Fig. 4 for a = 0.008 
and Hebbian noise. A very similar diagram is obtained 
in the case of full noise with b = v, and analysis of the 
power spectrum confirms the nature of the q-p states 
as non-stationary quasi-periodic solutions. In contrast, 
with initial overlaps mi = 0.21, m% — 7773 = 0.20, in the 
case of c = 3, the symmetric states become the only sta- 
ble states at low T for all values of v. Regions of stable 
Hopficld-like and cyclic states appear at higher T and, 
again, there appear non-stationary q-p solutions between 
these phases and the S phase. 

All components of the overlap in the H phase are differ- 
ent from zero for non-zero a at T ^= 0, although some of 
them may be small. In contrast, some components may 
be zero when T = 0, as in the case of finite loading |21j . 
Also, for non-zero a, the stable states in the C phase are 
still pcriod-4 cyclic solutions. We checked this explicitly 
for a = 0.008 and values for v and T within that phase. 

To see the overall role of extensive loading and to find 
the critical storage ratio a c for a given superposition of 
Hebbian and sequential learning, we consider next the 
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FIG. 5: Phase diagram for e = 4, extensive loading at T = 
and Hebbian noise, with initial overlaps mi = l,ni2 = ms — 
m 4 = 0. 

(a, v) phase diagram of stable states at T = for c = 4, 
shown in Fig. 5, for Hebbian noise and initial overlaps 
mi = l,m 2 = m 3 = m 4 = that favor single-pattern 
reconstruction. Note that the critical storage ratio for 
pure Hebbian learning, that is for v = 1, is a c = 0.269, 
in accordance with the known result for the layered net- 
work model [27j . Due to the v/{l — v) duality, it is also 
seen to be the critical ratio for pure sequential learning 
of the condensed patterns, as it should be, since a pure 
sequential noise is a Hebbian noise. The critical storage 
ratio a c for the retrieval of Hopfield-like or cyclic states 
is given by a point on the phase boundaries where the H 
or C phases end. In order to check our results obtained 
from numerical iterations, we performed numerical simu- 
lations to locate a few points on the first-order transitions 
between the ordered phases, at T = and we obtained 
results in good agreement. As will be seen below (cf. Fig. 
7) , somewhat different results are obtained in the case of 
full noise. Also, more symmetric initial conditions, say 
mi = 0.21, v%2 = TO3 = run = 0.20, lead to further q-p 
solutions. 

We consider next the (a, v) phase diagram of stable 
states at T = for c = 3 and Hebbian noise. For 
the initial overlaps mi = 1, 777,2 = 7713 = one ob- 
tains a similar diagram to that shown in Fig. 5 for 
c = 4. The first-order transitions between the H and 
the C phase with the SG phase are close to the case of 
c = 4. Instead, for more symmetric initial overlaps, say 
mi = 0.4, m2 = 0.3, 777,3 = 0.2, one finds a quite differ- 
ent phase diagram with a symmetric phase almost every- 
where below a transition to the SG phase and largely sup- 
pressed stable Hopfield-like and cyclic states, as shown 
in Fig. 6. A similar diagram, except for an enlarged S 
phase, is obtained in the case of full noise with b = v. 

We look now at the effects of Hebbian plus sequential 
noise for c = 4 and finite a at T = 0. We do this again 
for b = v and consider also the case of a matrix in which 
the high components have equally favored Hebbian and 
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FIG. 6: Phase diagram for c = 3, extensive loading at T = 
and Hebbian noise, with initial overlaps mi = 0.4, 7712 = 
0.3, m 3 = 0.2. 



sequential parts, in order to explore the symmetry of the 
model around b = 0.5, independently of v. In each case 
we consider two kinds of initial conditions: one favoring 
single pattern and cycle retrieval, with mi = 1,7772 = 
777.3 = m,4 = 0, and another more symmetric with 7711 = 
0.21,7772 = 7773 = 7774 = 0.20. 

The results for b = v and b = 0.5 for the first initial 
condition are shown in Figs. 7(a) and (b), respectively 
In both cases there is now a finite region of non-stationary 
q-p solutions below the phase of stable symmetric states, 
that was not present for purely Hebbian noise. This is a 
feature of complete (Hebbian plus sequential) noise and 
in the case of b — 0.5 there is also a considerable im- 
provement of the critical a c for Hopfield-like and cyclic 
retrieval. Indeed, we find the quite higher a c = 0.6438 
shown in Fig. 7(b). We also checked with other choices 
for b and found that b = 0.5 seems to be the optimal 
case. 

Coming back to the more symmetric initial condition 
for c = 4, one finds quite larger regions of stable symmet- 
ric states and of non-stationary q-p solutions and partic- 
ularly smaller regions of H and C phases for both b = v 
and for b = 0.5. 



V. DISCUSSION AND CONCLUSIONS 

We studied in this work the dynamics of the competi- 
tion between pattern reconstruction and asymmetric se- 
quence processing in an exactly solvable feed-forward lay- 
ered neural network model for both finite and for exten- 
sive loading of patterns. The strictly feed-forward nature 
of the network makes it an ideal system to study the dy- 
namics as a discrete-time evolution. Given the initial 
overlaps as inputs on the sites of the first layer, the dy- 
namics is generated by random local fields at every site 
on the next layer which depend on the synaptic intcrac- 
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FIG. 7: Phase diagram for c = 4, extensive loading at T — 0, 
and Hebbian plus sequential noise with initial overlaps mi — 
1, ni2 = m,3 = 7Ti4 = 0, for (a) b = v and (b) b = 0.5. 



tion between units on the two layers. In turn, the local 
fields determine the probability of updating of units at 
the sites of that layer. The asymmetry of the sequence 
processing in this work refers to the sequential part of the 
learning rule that consists of a synaptic interaction con- 
necting a pattern on a layer with the consecutive pattern 
in the sequence on the next layer. A symmetric sequence 
processing, not considered in this work, would involve an 
additional synaptic interaction of the same strength con- 
necting a pattern on a layer with the previous pattern in 
the sequence on the next layer. We come back to this 
issue below. 

The model studied here is based on the superposition 
of a Hebbian and a sequential learning rule for a finite 
cycle of condensed patterns and a stochastic noise due to 
a previously learnt macroscopic set of either single pat- 
terns that follow a Hebbian rule or a superposition of 
sequential patterns in a cycle with single patterns. The 
superposition we consider is suitable for the study of the 
competition between pattern reconstruction and asym- 
metric sequence processing. We were especially inter- 
ested in the effects of finite synaptic noise (a ^ 0) on the 
performance phase diagrams that appear in the case of 



extensive loading of patterns. Naively, one may expect 
that the form of the noise does not make a great differ- 
ence, as we found in most of this work, but this is not 
always the case. 

New dynamic equations for the overlaps and for the 
variance of the Gaussian noise were obtained in the form 
of discrete-time recursion relations for extensive load- 
ing of patterns. In the case of a Hebbian plus sequen- 
tial learning rule with a macroscopic number of patterns 
p = aN, the variance of the noise depends on a macro- 
scopic number of correlations between overlaps with non- 
condensed patterns, ((M^(l)M^(l))) . All these corre- 
lations could be relevant for the dynamics but when it 
comes to the fixed points (and stationary cycles) only 
a small number of them survives making the sequence 
processing a tractable and solvable problem, at least in 
the relevant cases we considered in this work. In all the 
cases we studied the correlations between overlaps with 
non-condensed patterns formed, practically, a finite and 
tractable set. 

Explicit phase diagrams of stable states and regions of 
non-stationary solutions were obtained in this work that 
show the effects of stochastic noise due to a macroscopic 
number of learnt patterns, and different behaviors may be 
obtained depending on a variety of relevant parameters. 
We also considered, briefly, the optimal case where b = 
0.5 in the synapses that generate the noise. The reason 
why this yields a larger a c for v = 1 or zero is that 
the symmetric phase is considerably enhanced towards 
extreme values of v combined with the fact that Hopfield- 
like retrieval in this model is, in general, more robust 
to stochastic noise than the stable symmetric states for 
small or large v. 

Some general conclusions that may be drawn from the 
phase diagrams are the following. First, as expected, 
the retrieval quality of Hopfield-like and cyclic states is 
gradually reduced with an increase in the storage ratio a. 
Less obvious, there is also a reduction in nearly single- 
pattern retrieval or in cycle retrieval as v or 1 — v are 
decreased, respectively. The change may be either to 
a symmetric state or to a spin-glass state. Eventually, 
depending on the form of the noise due to the previously 
learnt patterns, the transition could be to non-stationary 
quasi-periodic states. The symmetric states appear as a 
locking of the sequential transitions by the static Hebbian 
part of the learning rule. 

A further result of our work is that, for a Hopfield 
type initial condition m\ = 1,TO2 = = m± = 0, the 
first-order phase boundaries in the (T, v) phase diagrams 
where the Hopfield-like phase ends is practically the same 
for small a and cither Hebbian or complete noise, for both 
c = 4 and for c = 3. That is also the case for the (a, v) 
phase diagrams at T = 0. 

Turning now to predictions for somewhat larger, finite 
values of c, either even or odd for Hopfield type initial 
conditions, we expect that the non-stationary solutions 
that appear in the finite loading case for small T [2l| 
should disappear as soon as a becomes non-zero, allowing 
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for the presence of a symmetric phase down to T = 0, 
leaving a phase diagram with essentially no q-p solutions. 

It may be pointed out that there is a similarity be- 
tween our (a, v) phase diagrams at T = 0, with initial 
Hopfield type overlaps, and the corresponding phase dia- 
gram for pattern reconstruction and symmetric sequence 
processing referred to above obtained by means of sta- 
tistical mechanics |Tsj . The case of pure sequential pro- 
cessing is not considered in that work and, instead of a 
single-pattern retrieval region found there, we have here 
a Hopfield-like region (a difference already noted in the 
context of finite loading [2l]]). The boundaries of either 
of these regions have a similar dependence with the ratio 
— v). Moreover, there are similar regions of sym- 
metric phases and, in contrast to our work, there is also 
a region of correlated states. 

Based on the experience with the feed-forward layered 
network to process information with a Hebbian learning 
rule, one may expect qualitatively similar results to those 
obtained here to apply for a recurrent network with a 
Hcbbian-plus-scqucntial learning rule. Most of the work 
reported here deals with the dynamics near stationary 
states. Since our recursion relations for the overlaps and 
for the noise are general for the layered network model, 
they may also be employed to study the transients of 
the dynamics, in particular to find out what makes the 



correlations between distant (in pattern space) high com- 
ponents of the overlaps vanish. One may also study the 
slowing down of the dynamics due to the macroscopic 
number of non-condensed patterns. Work along some of 
these lines is currently in progress. 

As an extension of this work one may study the 
performance of this model with a number of sequences. 
There is recent work only with sequential learning, for 
an infinite number of limit cycles |26l| . We point out 
that there is renewed interest in sequence learning from 
a biological point of view [30l | and that asymmetric net- 
works have been found to be computationally superior 

m 
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